In parallel computing systems, such as High Performance Computing (HPC) applications, data storage systems must deal with the increasing amounts of data to be processed. As HPC environments grow to exascale (and larger) by becoming more distributed, sharded storage arrays comprised of a very large number of storage devices are expected to be employed. In sharded storage arrays, a user stores data on each storage device by first creating horizontally partitioned “shards” on each storage device. A serial stream of bytes is then typically written as stripes in a round-robin fashion across shards. In order to parallelize Input/Output (I/O) operations on the sharded storage arrays, it is desirable to have shards on a large number (if not all) of the available storage devices.
A need exists for improved techniques for reorganizing stored data according to user requests (e.g., based on a dimensional description or range information) to allow groups of sub-objects to be stored together, for example, to facilitate analysis tasks that require such groups of data.